ePBS Implementation - #496
Draft
JasonVranek wants to merge 86 commits into
Draft
Conversation
Adds the ePBS (gloas) builder-API surface and the strict header rulings, rebased onto main so it sits on top of the SSZ rewrite (#468) and the fork-from-slot fix (#487). ePBS endpoints: - getExecutionPayloadBid, submitBuilderPreferences, submitSignedBeaconBlock, with per-builder routing by SignedRequestAuth.data (no Eth-Builder-Url). - SSZ-first request/response with JSON fallback; the bid poll ladder honors the proposer's timing headers. Header discipline (builder-specs #165): - Eth-Consensus-Version required for JSON and SSZ on the request-auth endpoints; absent -> MissingVersionHeader, present-but-unsupported -> InvalidVersionHeader, both 400. The accepted set is Gloas only; any other fork name is a client bug. - A relay bid whose response fork is not Gloas on the Gloas-only endpoint is a bad relay response: dropped, never forwarded under the wrong fork. Status + observability: - zero addressed builders accepting an ePBS submission maps to 500, not 502 (neither endpoint declares 502); new PbsClientError::NoBuilderResponse. - decode/accept rejections counted in BEACON_NODE_STATUS via record_client_error; dropped relay responses in pbs_relay_invalid_response_total{reason,endpoint,relay_id}. Legacy PBS, the websocket get_header stream (#483), and the #487 fork fix are unchanged. Suite green, clippy clean.
… auth) builder-specs types/gloas/builder_preferences.yaml declares the container as [preferences, auth]; the struct had them reversed, so its SSZ encoding and tree-hash did not match the spec. Swap the fields and re-pin the spec-vector test to the canonical examples/gloas/builder_preferences_request.ssz fixed-part bytes.
A zero-length auth.message.data is invalid per builder-specs
types/gloas/request_auth.yaml (pattern {1,4096}) and addresses no builder. It
was only caught incidentally by the relay demux and could slip through a
catch-all relay match. Add a shared validate_auth_data check (EmptyAuthData ->
400) run first in both the bid and preferences auth validators.
External KM projection tooling round-trips auth_data through match_relays_by_auth_data, so pin the behaviors it relies on: userinfo and default-port are ignored when matching a relay entry URL, configured expected_auth_data takes exact-byte precedence per relay (cross-form collision with another relay's URL bytes yields the union set), and one auth_data may match several relays sharing the same configured bytes.
…anking clamp
beacon-APIs #630 makes the beacon node the enforcer on the ePBS path: it
MUST-rejects a bid below the per-key min_bid and values every bid at
value + min(execution_payment, max_execution_payment), clamping an
over-claimed trusted payment instead of rejecting it. CB's copies of
both checks therefore over-reject: a bid the BN would clamp-and-consider
or floor-check per key was dropped outright (worse still, the old
default cap of 0 rejected every nonzero execution payment). The legacy
get_header path keeps its own min_bid check (no per-key BN enforcement
exists there).
Since CB returns a single winner, the cap survives as a RANKING clamp so
CB's winner agrees with the BN's valuation: bids rank at
value + min(execution_payment, relay cap), where the per-relay
max_execution_payment_gwei overrides the global one. The config default
becomes u64::MAX (unclamped, matching the spec's MAX_EXECUTION_PAYMENT);
a 0 default would zero out every execution payment in ranking.
Dead plumbing removed: ValidationContext.{min_bid_gwei,
max_trusted_bid_gwei}, the validate_header_data params, the
TotalPaymentTooLow/TrustedBidTooHigh error variants, and the unread
HeaderInfo payment fields.
…nfig BuilderConfigDoc/BuilderEntryDoc follow keymanager-APIs builder_config: Uint64s as JSON strings, auth_data as 0x-hex, omitted fields off the wire. CanonicalDoc compares stored docs by value (hex decoded, entries sorted, Uint64 strings parsed) since the spec promises neither entry order nor hex case. The overlay carries the operational side (advertised_url, VCs, per-mux fallbacks) that does not belong in the CB fleet config.
…unknown mux keys Three config additions for the ePBS KM builder_config flow: - PbsConfig.advertised_urls: CB's externally-reachable URLs, consumed by the ePBS pipe self-URL guard (an unconfigured key's auth_data defaults to CB's own URL; the guard must recognize it instead of self-dialing). - MuxConfig.builder_boost_factor and MuxConfig.min_bid_eth (as_opt_eth_str, same codec as the global min_bid_eth): projection-only fields consumed by KM tooling that projects the mux config into per-key keymanager builder_config docs; the PBS runtime parses them, logs an INFO note at mux load, and otherwise ignores them. - warn-on-unknown [[mux]] keys: MuxConfig is shared with the legacy get_header path, so serde(deny_unknown_fields) would reject previously-valid configs; instead the config loaders make a best-effort pass over the raw TOML and WARN on any unrecognized mux key, giving typo visibility without a breaking change.
project() turns a CB mux config plus overlay into per-key KM docs: one entry per auth_data equivalence class (identical candidate bytes: expected_auth_data, else the configured relay URL with userinfo stripped by string surgery so no normalization touches the bytes), builder_pubkeys as the class union, entries sorted by (url, auth_data bytes). The raw TOML is parsed alongside cb-common's parser because RelayEntry holds a normalized Url while the auth_data convention wants the URL bytes exactly as configured. MuxProjectionFields is the seam for the upcoming MuxConfig builder_boost_factor/min_bid fields; the overlay per_mux map is the fallback source until then. File key-loaders resolve offline; HTTP and registry loaders are skipped with a warning. Enforced at projection time: 64-entry and 64-pubkey KM maxima, auth_data 1..=4096 bytes, per-class max_execution_payment_gwei agreement, no duplicate keys across muxes.
apply: per-VC preflight (authed keystores listing plus a builder_config probe on an enumerated key, so a 404 means missing keymanager-APIs #88 support rather than key-elsewhere), POST per projected key with 202/404/ 403 handling, exit non-zero unless each key is accepted by exactly one VC (zero acceptors errors; several is a duplicate-key slashing alarm). --dry-run prints, --emit writes per-key JSON plus a manifest, --prune POSTs {} (spec-equal to DELETE) for stored-but-unprojected keys. check: read-only canonical comparison with tiered findings and --fail-on; fields the projection omits resolve to VC config on GET and are skipped rather than flagged as drift.
axum mock VC covering: partitioned-key happy path (POST body equals the projected doc), 401 preflight, missing-#88 detection, zero-acceptor non-zero result, duplicate-key slashing alarm, 403 config-file-managed, prune body exactly {}, dry-run/emit sending nothing, canonical check (reordered entries + uppercase hex + VC-resolved fields produce no drift), drift + unroutable auth_data + duplicate-key findings.
An unmatched getExecutionPayloadBid no longer 400s outright: the decoded auth_data URL is dialed via a transient RelayClient through the shared send path, guarded fail-closed against CB's own advertised_urls. Bid sigverify is skipped per-context for the pipe relay only; the global skip_sigverify and configured-relay verification are unchanged.
…r URL Same demux, same semantics as the bid endpoint: empty match -> self-URL guard -> transient client through the shared send path, expecting the builder's 202.
…e lax-URL equivalent CB's url_matches ignores userinfo, host case and the default port, so two byte-distinct classes it treats as one relay would split builder_pubkeys across KM entries and the VC would silently reject the missing pubkey's winning bids. Conservative tool-local mirror; CB-side contract tests pin the real matcher.
v1 CB is a pure pipe: a stored builder URL outside the configured relays still routes, so it is operator-notable, not a failure.
…L output The log-settings pre-load went through CommitBoostConfig::from_env_path, so every service warned about each unknown mux key twice (once for the logs load, once for the primary config load). Route the logs load through a silent variant so each load path warns exactly once. Also skip serializing builder_boost_factor and min_bid_eth when None so a MuxConfig re-serialized to TOML round-trips cleanly.
…support A transport error or 5xx on the probe is its own error, never evidence of no support; an empty-keystore VC warns and POSTs anyway (the 404/405 responses answer the support question).
Url::join with an absolute path drops a /prefix base; concat on the trimmed base keeps it.
Group read (0o040) leaks the bearer token the same way world read does.
The mux_ext seam was written against pre-merge cb-common; now that builder_boost_factor and min_bid_eth exist on MuxConfig, the accessors read them directly (precedence: mux field > overlay > global).
Add optional PbsConfig fields gloas_fork_version (0x-hex, 4 bytes) and genesis_validators_root (0x-hex, 32 bytes) and thread them into the ePBS bid signature verification. Unset = the built-in constants, so existing configs verify exactly as before; devnets/networks whose gloas fork version or genesis root differ can now pass bid sigverify without skip_sigverify. Deriving these from the fork schedule is tracked separately.
Add projection-only PbsConfig fields min_bid_p2p_eth and builder_boost_factor_p2p (consumed by KM tooling, not the PBS runtime; INFO-logged when set). When set, the projected KM doc's KEY-LEVEL min_bid/builder_boost_factor come from them: projected entries always carry explicit per-entry values, so the key level only governs p2p bids and entries that omit their own. Unset keeps the uniform projection (key level = mux/global values) so existing docs are unchanged.
- Drop the auth-data and auth-signature span records on the bid path: they leak a bilateral secret if the field is ever declared and are no-ops today. The non-secret parent_root gains a router span field so its existing record surfaces. - url_matches canonicalizes a trailing-dot host so a fully-qualified name cannot slip the self-URL pipe guard. - check_gas_limit uses saturating_add so a near-max parent gas limit cannot overflow the adjustment check. - KmClient warns when a VC keymanager URL is plain HTTP to a non-loopback host, where the bearer token travels in cleartext. - cb-km check redacts secret-form (non-URL) auth_data to length-only in its findings, matching the apply-path redaction. - cb-km check diffs stored and projected entry sets both ways, itemizing each surplus stored entry instead of only reporting a count. - The signed-block fan-out spawns each send so a BN disconnect cannot cancel in-flight broadcasts, preserving the winner-accepts count.
cb-km inserted the relay URL's userinfo pubkey into each builder_config entry's builder_pubkeys. That pubkey is the relay's identity, not the builder's bid-signing key. Lodestar rejects any builder-API bid whose signing pubkey is not in builder_pubkeys (the check is skipped when the array is empty), so binding the relay identity silently rejected every bid. cb-km has no reliable way to learn the builder's actual bid-signing key from config, so emit an empty array (accept any builder for the key) until real support can supply that pubkey. Removes the now-dead AuthClass.builder_pubkeys field, its union collection, and the per-entry MAX_BUILDER_PUBKEYS cap (the const stays, still used by the --preserve-entries merge path). The auth_data and url were already userinfo-stripped, so no pubkey credential leaks there.
CB is a blind pipe for builder preferences: rejecting a stale or replayed submission is the builder's call, not the relay's. slot_has_passed rejected every preference whose slot had ended by the time CB validated it. On a live devnet the proposer submits an epoch ahead but the auth reaches CB after its slot, so the check rejected the whole preference stream: CB then served no header and the builder's bids only reached the proposer over p2p. Forward regardless of slot age; the builder rejects what it does not want. Removes the now-dead AuthSlotPassed error and the slot_has_passed unit tests.
ensure_no_lax_ambiguity aborted the whole cb-km projection when two builders on one host differed only by implicit-vs-explicit port, host case, or a supplied URL. Its premise -- that a lax URL collision would split builder_pubkeys across KM entries and make the VC reject winning bids -- is void: project_mux now emits builder_pubkeys empty unconditionally, so every entry accepts any builder and there is nothing to split. The two byte-distinct URLs simply project as two entries. Reintroduce only if real per-builder pubkeys are ever projected.
validate_header_data rejected any bid whose header fee_recipient differed from the operator-configured pbs.fee_recipient. On the gloas path this is wrong: the execution block's fee recipient is the builder's, and the proposer is paid via value + execution_payment, so header.fee_recipient is not expected to equal the proposer's configured recipient. An operator carrying a legacy fee_recipient would drop every legitimate builder bid. Bid validity is the builder's and the BN's job. Removes the check, its plumbing, and the now-dead FeeRecipientMismatch.
…opt-in
Block validity is the builder's job (builder-specs: an invalid block MUST be
rejected by the builder). By default CB now forwards the reveal bytes to every
builder without decoding them, so its decode strictness (SSZ over-read, JSON
deny_unknown_fields) no longer 400s a block the builder would accept. The new
strict_block_decode flag (default false) restores CB-side decode + a non-gloas
400 for operators who want it.
Two decode-path corrections that apply under strict mode and to the version
gate used in both modes:
- absent Content-Type now defaults to JSON, per the builder-specs preamble
("all requests by default send and receive JSON"); octet-stream is SSZ-only.
- Eth-Consensus-Version accepts gloas-or-later (the endpoint is "Gloas
onwards") and the outbound header carries the block's actual fork instead of
a hard-coded gloas; still exhaustive so a new fork forces an explicit add.
…buffer The old ePBS bid timeout was min(timeout_get_header_ms, late_in_slot - ms_into_slot), which returned 204 without contacting the builder once the request arrived past late_in_slot_time_ms (default 2s into the slot) -- a legacy mev-boost timing knob with no basis in the getExecutionPayloadBid spec, discarding bids the proposer would still accept. The beacon node sends its own deadline on every request (Date-Milliseconds + X-Timeout-Ms), so CB derives its timeout from that live value: it reserves proposer_deadline_buffer_ms (default 50) for the winning bid's return trip to the BN and the BN's own selection, and asks the builder for the rest. This replaces the drift-prone "configure deadline-minus-margin as an absolute" pattern with "configure the margin"; the BN's timeout can change and CB tracks it. On the ePBS path timeout_get_header_ms and late_in_slot_time_ms are no longer consulted (they remain load-bearing for the legacy get_header path, which carries no X-Timeout-Ms).
…-bottleneck-fixes
…pbs-bottleneck-fixes
The transient pipe forwards a bid/preferences request to the builder URL the proposer's signed auth_data names when no configured relay matches. It fails closed without advertised_urls: CB cannot tell an unconfigured key's self-URL default (which points at CB itself) from an external builder, so it will not dial -- a deliberate, SSRF-conscious opt-in. The defect was that it did so SILENTLY, rejecting with a bare 400 and no hint that one config field enables it. Now CB warns once at startup (when muxes are configured but advertised_urls is empty) and logs an actionable one-time hint on the first rejected pipe request. No behavior change: the pipe stays opt-in via advertised_urls.
The fee_recipient removal dropped the top-level Address import, but a mock in the test module still uses it. cargo check (non-test) did not flag it, so the cb-pbs test build broke. Import Address in the test module.
proposer_deadline_buffer_ms and strict_block_decode were added to PbsConfig but several test-only struct literals build it field-by-field, so the test build broke (cargo check does not compile the test cfg, so it was not caught earlier). Fill in the two fields in the signer, tests-utils, and cfg-file-update literals.
…ient tests Zero budget (the proposer's Date-Milliseconds + X-Timeout-Ms deadline, minus the buffer, has passed) means any bid would land too late for the beacon node to use, so return 204 without a doomed relay call. This honors the proposer's own deadline, not a CB late-in-slot cutoff, so it is not the preemptive skip the bid path used to do. Update the bid-path tests for the accepted behavior changes: fee_recipient is no longer enforced on the ePBS path (repurpose the wrong-fee-recipient test to assert the bid is served; drop the mux-fee-recipient test, which asserted the removed mux-level enforcement).
…ults /beacon_blocks decoding is now opt-in (strict_block_decode): the json-202, non-gloas-400, and unsupported-content-type-415 tests exercise the decode path, so enable strict mode in their configs. And builder preferences are no longer gated on slot age (blind pipe), so repurpose the slot-passed-400 test to assert a past-slot preference is forwarded and accepted.
Formats the ePBS bottleneck + transient-pipe work, and picks up five files with pre-existing drift under the pinned toolchain (types/mod.rs, signature.rs, apply.rs, mock_km.rs, mock_relay.rs).
# Conflicts: # Cargo.lock # Cargo.toml
…ipient knob - max_execution_payment_gwei is now Option<u64> (None = unclamped), mirroring the relay-level override. The old u64::MAX default was documented as a literal that TOML cannot parse, so copying it from config.example.toml bricked startup. - Add an unknown-field scanner for the [pbs] table (KNOWN_PBS_FIELDS): PbsConfig is flattened into StaticPbsConfig and cannot use deny_unknown_fields, so a typo'd knob (e.g. a security flag) parsed clean and silently stayed default. It now warns, matching the [[mux]] discipline. Wired on the default and custom-module load paths. - RelayConfig::validate rejects a zero bid_poll_timeout_ms / frequency_get_header_ms (a stall, not "unset"); proposer_deadline_buffer_ms is capped at one slot. - Document strict_block_decode in config.example.toml. - Remove fee_recipient: the field was read nowhere (the BN filters bids by fee_recipient for ePBS), yet its doc claimed CB rejected mismatched bids. Drop the field, the mux override, the dead trait accessor, and the tests that pinned non-enforcement.
In the default blind pipe CB forwards the reveal bytes unparsed, so the reveal must be SSZ. A JSON or otherwise non-SSZ reveal was relabeled octet-stream and failed opaquely at the builder (a 500 with no explanation). It is now rejected up front with 415. The Content-Type is resolved with the builder-specs default (JSON when the header is absent), so an unlabeled reveal is treated as JSON and rejected too, not assumed to be SSZ.
require_consensus_version_header accepted Gloas | Heze while its own doc said GLOAS ONLY. Narrow it to Gloas: a later fork's ePBS semantics are not yet validated here, so it is 400'd rather than silently handled as gloas. The match stays exhaustive with no wildcard, so a future lighthouse fork stops it compiling and forces an explicit decision to widen the set.
…g helper Remove comments that restate the adjacent, self-documenting code, correct two doc comments that no longer matched their code (the epbs_base_send_headers fork note and the mock received_auth field), fix a run-on test comment, and delete the unused content_type_encoding wrapper (only content_type_encoding_with_default is called).
…elay_id, shared client) The transient pipe dials a builder URL taken straight from untrusted auth data, so before building the client it now resolves the target and refuses any address in loopback / private / link-local / CGNAT (100.64.0.0/10) / v4-mapped-internal space, failing closed on a resolution error (PbsClientError::PipeTargetBlocked -> 400). A DNS-rebind (TOCTOU) between the check and the dial is a disclosed v1 limitation. The check is compiled and active in every build; only the `testing-flags` feature can skip it (a thread-local a test sets) so the e2e suite can dial a local mock. The pipe relay id is now a fixed "pipe" sentinel instead of the attacker-supplied URL host, which was an unbounded Prometheus relay_id cardinality vector. The pipe reuses a single shared reqwest::Client (built once in PbsState) instead of constructing a fresh one per bid - matching how configured relays already reuse their client and keeping the connection pool warm across slots. Adds RelayClient::with_client for that; resolve_addressed_relays / transient_pipe_relay are now async for the resolution step.
The proposer-supplied X-Timeout-Ms sets how long CB solicits a bid, but it had no upper bound: a large value pins an outbound relay connection open and grows the timing-games poll ladder for that whole duration. Cap the budget at one slot (chain-aware) before deriving the timeout, bounding the duration a single request can hold a relay connection and the ladder depth.
…dvertised_urls `cb-km check` now flags the self-dial misconfiguration the docs warn about: every advertised_url km-tool projects is echoed by a VC in its auth data, decoded by CB, and must be recognized as CB's own via [pbs] advertised_urls. If advertised_urls is set but does not cover a projected URL, a bid addressed to it decodes to CB's own URL and self-dials recursively. Since the operator runs `cb-km check` as the gate before apply, an uncovered URL is an error. Mirrors cb-pbs `url_matches` locally with a drift-guard test.
The three ePBS handlers carried the same error arm (4xx warns, 5xx errors, count the status) and the two write routes the same spawn-and-join fan-out. Both move to cb-pbs utils as `record_request_failure` and `join_detached_sends`; `expect_status` is inlined into its only caller. submitBuilderPreferences encodes the SSZ body once and shares it across the sends instead of cloning and re-encoding the request per relay. submitSignedBeaconBlock no longer re-stamps Eth-Consensus-Version with the parsed fork: the header validator accepts Gloas only, which is the value the base headers already carry.
…eader getExecutionPayloadBid parsed the Accept header twice: once for the response encoding and again to build the relay-side Accept. It is parsed once and threaded through; `preferred(&[Ssz, Json])` always equals the primary, so `encode_bid_response` takes an `EncodingType` and its unreachable 406 arm goes. The relay Accept is one of two static headers, so `build_outbound_accept` and its q-value ladder (only ever fed two entries) are deleted, with the JSON-first static pinned by a test like its SSZ-first twin. Also on the bid path: the request start time returned by the single poll was never read, and was the only reason `select_max_bid` was generic over its label; `ms_into_slot` and the per-relay ranking cap are computed once. The handler adopts the shared failure arm.
The variant held a u16 that `status_code` converted back with a 502 fallback its only constructor could never reach. It holds the `StatusCode` directly.
`load_pbs_config` and `load_pbs_custom_config` carried the same advertised_urls warning; it moves to one helper, and the runtime once-warn in the pipe stops restating it. km-tool's `resolve_min_bid` is inlined at its only call site.
The bid suite hand-rolled 23 PBS boots and the preferences suite two; they run through `spawn_mock_relay` + `setup_pbs`, which the existing `setup_relay*` helpers now delegate to. The default JSON bid request and the hand-built spec URL each get one helper, the proposer pubkey literal becomes `TEST_PROPOSER_PUBKEY`, and two clippy nits in the rewritten file are fixed. The mock relay computes its served fork and signing key once.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.